Papers with speech-only baselines
Seeing is Believing: Emotion-Aware Audio-Visual Language Modeling for Expressive Speech Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | AVLM integrates full-face visual cues into a pre-trained expressive speech model. |
| Approach: | They propose an Audio-Visual Language Model (AVLM) for expressive speech generation by integrating full-face visual cues into a pre-trained expressive speech model. |
| Outcome: | The proposed model incorporates full-face visual cues into a pre-trained expressive speech model. |
Exploring the Potential of Multimodal LLM with Knowledge-Intensive Multimodal ASR (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in multimodal large language models have made significant progress in integrating information across various modalities, yet real-world applications in educational and scientific domains remain challenging. |
| Approach: | They propose a task that focuses on transcribing scientific conference videos by leveraging visual information from slides to enhance the accuracy of technical terminologies. |
| Outcome: | The proposed framework improves transcript quality through post-editing and improves performance over speech-only baselines. |